Data Validation Process

• Data validation entails the assessment and verification of data quality, accuracy, and reliability, thereby supporting informed decision-making.

• Within World Justice Project (WJP), we have identified three distinct processes where these data checks are utilized:

  • The data cleaning process.
  • The estimation of results.
  • The analysis of results.

• Currently, our focus lies on developing protocols for the analysis of results.

Data cleaning process

• The data cleaning process involves utilizing data validation to ensure the reliability of observations.

• In the case of WJP, specific variables are included and cross-referenced with external sources and the Rule of Law Index to ensure data consistency and reliability.

• Challenges:

  • What level of analysis will be used for data checking? Will it be at the country level or the NUTS level?
  • Is the process of comparison with external sources necessary in the cleaning process? With which sources would it be compared?

Estimations of results

• The estimation process involves data validation using two approaches: replication checks and data consistency assessment.

• Replication checks: Every estimated data point is independently replicated by another data analyst to ensure accuracy and reliability.

• Data consistency assessment: Observations that lead to significant changes are meticulously reviewed, especially in the case of expert surveys. Additionally, aggregate responses from population surveys are assessed to ensure data consistency.

• Challenges:

  • Determining the selection of appropriate third-party sources for data estimation and assessment.
  • Identifying the specific stages of the process where data analytics can intervene to enhance the overall process.

Analysis of results

• After gathering all the results to be included, it is essential to conduct a comprehensive validation of these findings.

• The variations among the results are carefully examined to detect any biases or inaccuracies in any given country.

• The results are then compared with other sources to ensure their consistency and reliability.

• Challenges:

  • Can the questions be cross-referenced with the specified NUTS regions?
  • If we decide to aggregate the scores by country, how should we determine the weighting of each component?
  • How should we address situations where external data for certain sub-pillars is unavailable for comparison?

Introduction

• The protocol primarily focuses on validating the results obtained, rather than directly validating the quality of the underlying data.

• This protocol serves as a rigorous quantitative complement and a valuable tool for post-estimation validations.

  • It does not replace the checks conducted by researchers.

Methodology

• These methodologies validate WJP Global Index results and reports for Latin America and the Caribbean.

• Quantitative methods detect data changes and discrepancies with other sources.

• They complement qualitative research but do not replace it.

• Methodologies evaluate results, not data collection process.

• Two types of methodologies: internal and external validations.

Internal and External Validations

• Internal validations: These checks provide confidence in the data quality, ensuring it is representative, consistent, and free from distortions or biases.

  • Outliers’ detection

  • Changes over time

• External validations: These checks provide valuable insights into the validity and credibility of the project results, ensuring they are in line with established standards and corroborated by independent measures

  • Rankings and other indexes

  • Third party sources questions

Internal Validations

External Validations

Projects

• WJP Rule of Law Index

  • The Index Team has devised a protocol to validate the data and results for 140 countries worldwide. This protocol contains some of the methodologies presented.

• Latin America and the Caribbean Reports

  • In the case of reports focusing on Latin America and the Caribbean, validations were conducted at the question level, employing most of the methodologies presented.

• European Union Subnational Project

  • The Data Analytics Team proposes a harmonized data validation approach for the European Union report, combining methodologies used in both the index and the reports.

EU Subnational Project

• EU subnational project aims to generate indicators for assessing justice, governance, and the rule of law in 110 regions across 27 EU member states.

• Three resources (QRQ, GPP, third-party sources) utilized to develop indicators for NUTS and country-level scores.

• Comparing results with other sources is crucial for reliability and confidence assessment.

• Data Analytics Unit implements a protocol integrating methodologies to validate and compare project outputs.

• The protocol complements qualitative checks, ensuring accuracy, identifying inconsistencies, and instilling confidence in the obtained results.

Integrated Process of Validation

• At pillar level

  • An examination of outliers at the NUTS level for each pillar.
  • A comparison of the aggregated pillar scores by country with rankings from external sources relevant to the respective pillar.

• At sub-pillar level

  • An assessment of outliers at the NUTS level for each sub-pillar.
  • Whenever possible, a comparison of the aggregated sub-pillar scores by country with rankings from external sources associated with the specific sub-pillar.
  • An analysis of changes in the selected questions over time within each sub-pillar.
  • A comparison of the questions with external sources.

Possible Outcomes

• Insights report

  • We plan to generate a comprehensive report highlighting the key insights from this analysis.
  • This report will include the decisions made and recommendations regarding the outcomes.

• Web platform to test the outcomes

  • We will design an application that facilitates the external and internal testing of all outcomes on a country-specific basis.
  • This app will integrate qualitative checks and display all the flags associated with each pillar and sub-pillar within the platform.

Resources

Progress

• Completed development of the conceptual framework and questionnaire.

• Almost final version of the catalog.

• Identified a comprehensive list of potential sources for comparison for each pillar and sub-pillar.

• Successfully implemented codes and step-by-step instructions for all internal tests.

• Achieved significant progress in designing the desired outcomes, with examples of outcomes serving as initial reference points.

Gaps

• Identify new sources for sub-pillars with insufficient information to initiate external comparisons.

  • Researchers’ assistance is required to align these sources with the conceptual framework.

• Develop data cleaning functions for the selected external sources.

• Establish a threshold for data comparison.

• Integrate qualitative checks into the quantitative analysis.

• Begin designing the platform.

Questions